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DETAILED ACTION 
Claim Rejections '35 use §101 

1. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or 
any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and 
requirements of this title. ' 

2. Claims 1-4, 6, 7 and 9-13 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed to non-statutory subject matter. 

3. Claims 1-4, 7, 9, and 10 appear to be to an abstract idea rather than a practical application 
of the idea. Claims 1-4, 7, 9, and 10 does not result in a physical transformation nor do they 
appear to provide a useful, concrete and tangible result. Specifically, it does not appear to 
produce a tangible result because merely computing the positions of annotations and 
synthesizing images are nothing more than thoughts or computations within a processor. The 
practical application is not explicitly recited in the claims nor does it flov^^ inherently therefrom. 
Claim 7 recites "displaying the read actually taken image"; however never makes use of the 
result of the calculations. Annex 5 of the "Interim Guidelines for Examination of Patent . 
Application for Patent Subject Matter Eligibility" provides guidance with respect to the 
determination of the patentability of mathematical algorithms. If the "acts" of a claimed process 
manipulate only numbers, abstract concepts or ideas, or signals representing any of the 
foregoing, the acts are not being applied to appropriate subject matter. Benson, 409 U.S. at 71- 
72, 175 USPQ at 676. Thus, a process consisting solely of mathematical operations, i.e., 
converting one set of number into another set of numbers does not manipulate appropriate 
subject matter and thus cannot constitute a statutory process. 
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4. Claims 6 and 1 1 are directed to computer programs. Computer programs, per se, are not 
in one of the statutory categories of invention (See MPEP § 2106 with regard to computer 
programs.). Functional descriptive material claimed in combination with an appropriate 
computer readable medium to enable the functionality to be realized is patent eligible subject 
matter if it is capable of producing a useful, concrete and tangible result when used in the 
computer system. However, the control program recited in claims 6 and 1 1 does not produce a 
useful, concrete and tangible result when used in the computer system. A process consisting 
solely of mathematical operations, i.e., converting one set of number into another set of numbers 
does not manipulate appropriate subject matter and thus cannot constitute a statutory process (see 
the preceding remarks with respect to claims 1 and 7). Furthermore, the claims are directed to the 
program itself and not the medium encoded with the program. Therefore, claims 6 and 1 1 are 
directed to non-statutory subject matter. 

5. Claims 12 and 13 preempt an abstract idea. A claim may not preempt every substantial 
practical application of an abstract idea, law of nature or natural phenomena because it would in 
practical effect be a patent on the judicial exceptions itself Claims 12 and 13 appear to be 
directed to a generic computing system executing a method. 

6. To expedite a complete examination of the instant application, the claims rejected under 
35 U.S.C. 101 as non-statutory subject matter are further rejected as set forth below in 
anticipation of applicant amending the claims to place them within the four categories of 
invention. 
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Claim Rejections - 35 USC § 103 

7. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

8. The text of those sections of Title 35, U.S. Code not included in this action can be found 
in a prior Office action. 

9. Claims 1, 3, 6 and 12 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Masakatsu Kourogi, Takeshi Kurata, Katsuhiko Sakaue, and Yoichi Muraoka, ^^A 
panorama-based technique for annotation overlay and its real time implementation,'^ July 
30, 2000, Proceedings of IEEE International Conference on Multimedia and Expo, p. 657- 
660 (Kourogi et al) in view of U.S. Patent No. 6,064,399 to Teo. 

10. With regard to claim 1, Kourogi et al discloses "An information processing method 
comprising: 

. a, a viewpoint position / sight line direction determination step of determining a 
viewpoint position and a sight line direction on a map (2^^ paragraph of section 2: 'The 
referred panoramic image will be switched if necessary as the camera moves around... By 
tracking which panorama is referred, we can also estimate the position and trajectory of 
the user. "; Figure 8 shows an Estimation of the user 's position; Figure 9 shows 
Estimation of the user 's orientation; Figure 2 shows a map); 

b. a reading step of reading a panoramic image according to the viewpoint position 
from a storage unit which stores plural panoramic images (Figure 2 shows ''the 
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panoramic image used is switched while the user moves"), and cutting out a display 
image from a storage unit which stores plural panoramic images , and cutting out a 
display image based on the sight line direction (r^ paragraph of section 3,2: "„.aset of 
four panoramic imases were acquired at point [sj A-D in Figure 5... Figure 6 shows a 
set of panoramas; Figure 7 shows output frames that are *'cut out" of the panoramic 
image data; 2nd paragraph of section 2: "The referred panoramic image will be switched 
if necessary as the camera moves around... By tracking which panorama is referred, we 
can also estimate the position and trajectory of the user. Figures 8 and 9 show , 
estimating the users sight line direction, which is equivalent to position and orientation), 
wherein the storage unit stores the plural panoramic images respectively while associated 
them with routes set on the map (2"^^ paragraph of section 1: ''Experimental results show 
that this method can, with low cost PCs locate and orient input frames and display the 
frames.., "; Figure 5 shows a route on a map associated with panoramic image data sets 
A'D); 

c. a combining step of combining an annotation object with the display image based 
on the calculated display position (r^ paragraph of section 3.3: '-In the experiments the 
user moved along the path A -^B -^D (Figure 5). The output video frames 
overlaid with annotations are shown in Figure 7, and the user 's estimated position and 
orientation are shown in Figure 8 and 9 
1 1 . Although Kourogi et al teaches using the PCs for "locating," "searching" and 
"compressing" image data, Kourogi et al does not use the language "storing". However, this 
feature is deemed to be inherent to the PCs disclosed as section 3 shows implementing the 
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method on a "PC cluster" (section 3.1) and "searching" the set of image data (section 3.3). The 
Kourogi et al system would be inoperative if the panoramic image data were not stored. 

12. Kourogi et al does not expressly disclose "a relative coordinate calculating step of 
calculating relative coordinates of an annotation object with respect to the route to which the 
view position corresponds" or a "display position calculating a display position of said 
annotation object based on the viewpoint position and the relative coordinates of said annotation 
object." 

13. Teo discloses "a relative coordinate calculating step of calculating relative coordinates of 
an annotation object with respect to the route to which the view position corresponds" {lines 44- 
47 of column 8 (emphasis added): "These four vectors all emanate from a viewins position in the 
center of the scene, and this position is taken to be the orisin of a three-dimensional coordinate 
system. The present invention finds appropriate positive scalar s pi, P29 pj and P4S0 that the 
scaled vectors pi vy, p2 V2, p3 vj and p4 V4 form the vertices of a planar parallelogram.., "); 
"display position calculating step a display position of said annotation object based on the 
viewpoint position and the relative coordinates of said annotation object" {lines 31-36 of column 
8: "Once the user has positioned the quadrilateral 430 as desired, then the image or video 
segmented to be composited onto the panorama can be correctly embedded. Specifically, a linear 
transformation that transforms the rectangular frame of the image or video onto the frame of the 
planar parallelogram is determined"; lines 19-20 of column 10: ''The resulting (6,^) pairs map 
directly into pixel locations in the digital panorama. "); and "a combining step of combining an 
annotation object with the display image based on the calculated display position" {}ines 34-48 of 
column 7: " Image portion 310 is corrected for view perspective, and as such appears natural 
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The embedded wall picture 310, being an integral part of the panorama, is also corrected for 
view perspective, "; Figure 3) 

14. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to incorporate "a relative coordinate calculating step," "display position calculating step," 
as taught by Teo in the system disclosed by Kourogi et al. The motivation for doing so would 
have been to improve the aesthetic quality of the annotated images, as the method disclosed by 
Teo makes inserted graphic data appear "natural" to the user. Therefore, it would have been 
obvious to combine Teo with Kourogi et al to obtain the invention specified in claim 1 . 

15. With regard to claim 3, Kourogi et al discloses "each said route is defined by segment 
points ends thereof (figure 5 shows a route with segment points A-D). Kourogi et al does not 
expressly disclose "said relative coordinate calculating step sets one segment point of the route to 
which the view position corresponds as the origin, and sets said route as an x axis." Teo discloses 
"said relative coordinate calculating step sets one segment point of the route to which the view 
position corresponds as the origin {lines 44-47 of column 8 (emphasis added): 'These four 
vectors all emanate from a viewins position in the center of the scene, and this position is taken 
to be the origin of a three-dimensional coordinate system. '\ Teo discloses a y-axis in the 
vertical direction (lines 44-46 of column 9: Vn the ensuing description a frame of reference is 
used in which the second of the three Cartesian coordinates, the y-coordinate, points in the 
vertical direction. "). One of ordinary skill in the art would recognize that the Cartesian 
coordinate system recited in lines 44-46 of column 9 uses three perpendicular axes x, y and z. At 
the time of the invention, it would have been obvious to a person of ordinary skill in the art to set 
the route as the x-axis in the system and method disclosed by the combination of Kourogi et al 
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and Teo. One of ordinary skill in the art would have been motivated to do so since the user is 
permitted to move in the horizontal direction in the panoramic image viewer disclosed Kourogi 
et al and the route would naturally correspond to a horizontal axis. Therefore, it would have been 
obvious to further modify the combination of Kourogi et al and Teo to obtain the invention 
specified in claim 3. 

16. Claim 6 recites the limitations of claim 1 as a "control program for causing a computer to 
execute a information processing method." As previously shown, the combination of Kourogi et 
al and Teo shows the limitations recited in claim 1 . Furthermore, Kourogi et al discloses a 
"control program stored on a computer-readable medium for causing a computer to execute a 
information processing method" (section 3: 'We implemented our method as software running 
on a PC cluster and evaluated it in experiments ..."). 

17. Claim 12 recites the limitations of claim 1 as a "an image reproduction apparatus." As 
previously shown, the combination of Kourogi et al and Teo shows the limitations recited in 
claim 1. Furthermore, Kourogi et al "an image reproduction apparatus" (section 3: *We 
implemented our method as software running on a PC cluster and evaluated it. in experiments 
which a user was equipped with a wearable display, a wearable camera, and a wearable 
computer. 

18. Claims 2 and 4 are rejected under 35 U.S.C, 103(a) as being unpatentable over 
Masakatsu Kourogi, Takeshi Kurata, Katsuhiko Sakaue, and Yoichi Muraoka, ^^A 
panorama-based technique for annotation overlay and its real time implementation/' July 
30, 2000, Proceedings of IEEE International Conference on Multimedia and Expo, p. 657- 
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660 (Kourogi et al) in view of U.S. Patent No. 6,064^99 to Teo in view of U.S. Patent No. 
6,563,529 to Jongerius. 

19. With regard to claim 2, Kourogi et al discloses "a two-dimensional map" in Figure 2 and 
Figure 5. The combination of Kourogi et al and Teo does not expressly disclose the map is an 
image. Jongerius discloses a panoramic image from a viewpoint on a given map displayed along 
with an image of the two-dimensional map image {Figure 4\ lines 1 7-22 of column 5: "As will be 
explained, detailed field of view 40 shows a new area of the panoramic view of FIG. 1 slightly to 
the left of the view of FIG, 2, while map 38 shows a new highlighted view area 36, rotated 
slightly counter-clockwise (CCW) from view area 34 of FIG, 2. "). 

20. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to incorporate a two-dimensional map image as disclosed by Jongerius in the method and 
system disclosed by Kourogi et al. The motivation for doing so would have been to provide the 
user with a better understanding of the area being viewed. Therefore, it would have been obvious 
to combine Kourogi et al and Teo with Jongerius to obtain the invention specified in claim 2. 

21 . With regard to lines 1-5 of claim 4, Teo discloses said relative coordinate calculating step 
acquires observation directions of said annotation object, and calculates the relative coordinates 
of said annotation object from the observation direction {Figure 5; lines 44-47 of column 8 
(emphasis added): "These four vectors all emanate from a viewing position in the center of the 
scene, and this position is taken to be the origin of a three-dimensional coordinate system. The 
present invention finds appropriate positive scalars pi, P2, p3 cind p4S0 that the scaled vectors p\ 
V/, P2 V2» Pj vj and p4 V4 form the vertices of a planar parallelogram.,. lines 31 -36 of column 
8: "Once the user has positioned the quadrilateral 430 as desired, then the image or video 
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segmented to be composited onto the panorama can be correctly embedded. Specifically, a linear 
transformation that transforms the rectangular frame of the image or video onto the frame of the 
planar parallelogram is determined. lines 19-20 of column 10: ''The resulting (6,^) pairs map 
directly into pixel locations in the digital panorama, Teo does not expressly disclose doing so 
"at respective segment points." Kourogi et al disclose calculating the position of annotation 
objects at respective segment points {1'^ paragraph of section 3 J: ''In the experiments the user 
moved along the path A ->B ->C -^D (Figure 5), The output video frames overlaid with 
annotations are shown in Figure 7, and the user 's estimated position and orientation are shown 
in Figure 8 and 9"), At the time of the invention, it would have been obvious to a person of 
ordinary skill in the art to apply the method disclosed by Teo at the respective segment points in 
Kourogi et al. The motivation for doing so would have been to improve the aesthetic quality of 
the annotated images, as the method disclosed by Teo makes inserted graphic data appear 
"natural" to the user. 

22. With regard to lines 6 and 7 of claim 4, Kourogi et al discloses allowing "the user's 
camera to move around the environment" (see Figure 2.4) and setting an observation direction 
(see Figures 8 and 9); however, Kourogi et al does not expressly disclose "said information 
processing method further comprises a setting step of manually setting the observation directions 
at the segment points by using a GUI (graphical user interface)." Jongerius discloses said 
information processing method further comprises a setting step of manually setting the 
observation directions by using a GUI (graphical user interface) {Figures 4-6 show a GUI; lines 
59-65 of column 5: ''As stated, movement of the field of view can be effected by dragging the 
cursor in the detailed view of FIG. 2, or in the detailed view window 40 of FIG, 4. Movement of 
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the field of view can also be effected by clicking and dragging the cursor in the map of FIG. 3 or 
map window 38 of FIG. 4. /.e., the user would position the cursor anywhere in the map of FIG. 
i, or anywhere in map window 38 of FIG. 4. 

23. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to incorporate a GUI as disclosed by Jongerius in the system and method disclosed by 
Kourogi et al. The motivation for doing so would have been to allow the user to intuitively 
control the system. Therefore, it would have been obvious to further modify the combination of 
Kourogi et al and Teo with Jongerius to obtain the invention specified in claim 4. 

24. Claims 7, 11 and 13 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
U.S. Patent No. 6,563,529 to Jongerius in view of Masakatsu Kourogi, Takeshi Kurata, 
Katsuhiko Sakaue, and Yoichi Muraoka, ^^A panorama-based technique for annotation 
overlay and its real time implementation,'' July 30, 2000, Proceedings of IEEE 
International Conference on Multimedia and Expo, p. 657-660 (Kourogi et al). 

25. With regard to claim 7, Jongerius discloses "an information processing method used in an 
image reproduction apparatus for achieving walk-through in a virtual space represented by using 
an actually taken image (lines 30-34 of column 5: " This will cause the portion of the panoramic 
image of FIG. 1 shown in FIG. 2 to change and move, just as if the user actually stood at 
location 30 and turned to the right or left. "), said method comprising the steps of: 

d. reading the actually taken image from a storage unit which stores plural actually 
taken images respectively {lines 16-19 of column 7: 'Then a 360^ panoramic image, such 
as the one of FIG, 7, is installed on the computer's hard disk in the working directory 
under the name panoramic.jpg, in JPEG format. ") being made correspondent to a map, 
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and displaying the read actually taken image (lines 44-49 of column 2: "„Jo provide such 
a viewer in which the detailed image and the map image are never be out of sync because 
any change in the detailed image is immediately reflected in the map image, and any 
change in the map image is immediately reflected in the detailed image.,. 

26. Jongerius does not expressly disclose "synthesizing an annotation image to the actually 
taken image" or "setting an annotation display position in the displayed actually taken image 
based on a manual instruction; and calculating an annotation display position of another actually 
taken image located between the plural actually taken images to which the annotation display 
positions have been set respectively, based on the annotation display portions respectively set in 
the plural actually taken images. 

27. Kourogi et al discloses "synthesizing an annotation image to the actually taken image" 
and "setting an annotation display position in the displayed actually taken image based on a 
manual instruction (2'''^ paragraph of section 1: 'The proposed method uses (1) a set of 
panoramic images acquired at various points in the environment, (2) annotations attached to the 
panoramas and (3) neighborhood relationships between panoramas as prior knowledge about 
the environment. "; 1st paragraph of section 2 (emphasis added): **Our approach uses a 
panoramic image to which annotations are manually attached ... 4^^ paragraph of section 3.1: 
" .,.we used a small CCD camera to capture input video frames... and calculating an 
annotation display position of another actually taken image located between the plural actually 
taken images to which the annotation display positions have been set respectively, based on the 
annotation display portions respectively set in the plural actually taken images {1^^ paragraph of 
section 2: "Our approach uses a panoramic image to which annotations are manually attached 
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as the source of information. When an input frame is given, it is aligned with the referred 
panoramic image. Then, we can map the positions of annotations from the panorama to the 
frame... 7^' paragraph of section 2: " Then, we can map the positions of annotations from the 
panorama to the frame, and thus can create the frame overlaid with the annotations as shown in 
Figure 1. "). 

28. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to incorporate synthesizing annotation images as disclosed by Kourogi et al in the method 
and system disclosed by Jongerius. The motivation for doing so would have been to make the 
image more informative by providing the user with additional textual or graphical information 
about a specific object depicted in the panoramic image data. Therefore, it would have been 
obvious to combine Kourogi et al with Jongerius to obtain the invention specified in claim 7. 

29. Claim 1 1 recites the limitations of claim 7 as a "control program for causing a computer 
to execute a information processing method." As previously shown, the combination of 
Jongerius and Kourogi et al shows the limitations recited in claim 7. Furthermore, Jongerius 
discloses "control program stored on a computer-readable medium for causing a computer to 
execute a information processing method" {see line 66 of column 6 through line 15 of column 7). . 

30. Claim 13 recites the limitations of claim 7 as a "an image reproduction apparatus." As 
previously shown, the combination of Jongerius and Kourogi et al shows the limitations recited 
in claim 7. Furthermore, Jongerius discloses "an image reproduction apparatus" {see line 66 of 
column 6 through line 15 of column 7). 

31. Claims 9 and 10 are rejected under 35 U.S.C. 103(a) as being unpatentable over U.S. 
Patent No. 6,563,529 to Jongerius in view of Masaiiatsu Kourogi, Takeshi Kurata, 
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Katsuhiko Sakaue, and Yoichi Muraoka, ^^A panorama-based technique for annotation 
overlay and its real time implementation," July 30, 2000, Proceedings of IEEE 
International Conference on Multimedia and Expo, p. 657-660 (Kourogi et al) in view of 
Erwin Pang, and Dimitrios Hatzinakos, ^'An Efficient Implementation of Affine 
Transformation Using One-Dimensional FFT's," April 1997, Proceedings of 1997 IEEE 
International Conference on Acoustics, Speech, and Signal Processing, Vol. 4, p. 2885-2888 
(Pang et al). 

32. With regard to claims 9 and 10, Kourogi et al discloses in the first paragraph of section 
2.1: 

We estimate image alignment parameters between an input frame and a referred 
panoramic image, by using a fast and robust gradient-based method that can find affine 
or projective parameters of image alignment between images. We use affine model 
because of its stability of estimation. 

33. The combination of Jongerius and Kourogi et al does not expressly disclose "said 
calculating step acquires the annotation display position of another actually taken image, by 
interpolating the annotation display positions respectively set in the plural actually taken images 
wherein the interpolation is non-linear interpolation" as recited in claims 9 and 10. Pang et al 
teaches in the first paragraph of section 1 (emphasis added): 

Geometrical transformation of digital image is a common application in image 
processing [I] [2]. The uniformly distributed samples on a two-dimensional plane, after 
being transformed, will be misaligned with the reference grid pattern which are only 
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defined for discrete locations. An interpolation of these transformed pixels is thus needed 
to recover those on the grid points. 

34. Pang et al further discloses, "Cubic interpolation [1 1] is more accurate since it uses more 
[neighboring] points for resampling" {1^^ paragraph of section 4.1), 

35. At the time of the invention, it would have been obvious to a person of ordinary skill in 
the art to incorporate non-linear interpolation for determining the annotation position as taught 
by Pang et al in the system disclosed by Kourogi et al, and. to do so from among plural non-linear 
curves previously held. The motivation for doing so would have been to "recover the grid points 
in the image" after the affine transformation is applied, as suggested by Pang et al, in a 
computationally efficient manner. Therefore, it would have been obvious to combine Pang et al 
with Kourogi et al to obtain the invention specified in claims 9 and 10. 

Response to Arguments 

36. Applicant's arguments with respect to the pending claims have been considered but are 
moot in view of the new ground(s) of rejection. It should be noted that in the Office action dated 
6/6/2006 claims 1-12 were rejected under 35 U.S.C. 101. 

Conclusion 

37. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. Ismo Rakkolainen, Jani Timmerheid, Teija Vainio, "A 3D City Info for Mobile 
Users," November 9, 2000, Proceedings of the 3rd International Workshop on Intelligent 
Interactive Assistance and Mobile Multi-Media Computing, p. 1 15-121 shows a three- 
dimensional view and a corresponding map view. U.S. Patent No. 6,968,973 to Uyttendaele et al 
discloses an annotated image-based virtual tour. 
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38. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 . 1 36(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jason M. Repko whose telephone number is 571-272-8624. The. 
examiner can normally be reached on Monday through Friday 8:30 am -5:00 pm. 

If attempts to reach the examiner by telephone are unsuccessfiil, the examiner's 
supervisor, Ulka Chauhan can be reached on 571-272-7782. The fax phone number for the 
organization where this application or proceeding is assigned is 571-273-8300. 
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Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for impublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would 
like assistance from a USPTO Customer Service Representative or access to the automated 
information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. 



JMR 



*"'ULKA CHAUHAN 
SUPERVISORY PATENT EXAMINER 




